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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 
CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 

1 .17(e) has been timely paid, the finality of the previous Office action has been withdrawn 
pursuant to 37 CFR 1.1 14. Applicant's submission filed on 12/07/2009 has been entered. 

Response to Amendment 

2. The amendments, filed on 12/07/2009, have been entered and made of record. Claims 1-8, 
10-19, 21-30 and 32-36 are pending. 

Response to Arguments 

3. Applicant's arguments with respect to claims 1-8, 10-19, 21-30 and 32-36 have been 
considered but are moot in view of the new ground(s) or rejection. 

Claim Rejections - 35 USC § 102 

4. The text of those sections of Title 35, U.S. Code not included in this action can be found in a 
prior Office action. 

5. Claims 1-3, 5, 14-16, 25-27, and 36 are rejected under 35 U.S.C. § 102(b) as being 
unpatentable over Ishikawa (US 6,549,650). 
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Regarding claim 1 , Ishikawa discloses a method for generating a wide image video 
sequence (moving panoramic image) using a device (a binocular image sensing apparatus 
6001, Fig 38) having at least two video cameras substantially co-located in a predetermined 
relationship to each other (right and left image sensing optical systems 6004b and 6004a, Fig 
38) such that there will be an overlap between images from the respective cameras 
(synthesizing the right and left images 6201b and 6201a into a single panoramic image, an 
overlapping region fijr joining the two. Fig 40), said method comprising the steps of: 

a. forming a synthetic image fi-om images of the respective cameras (two images 
6201b' and 6201a' are synthesized into a single panoramic synthesized image. Fig 
40) by (i) identifying corresponding parts in overlapping parts of the images (the 
two images 6201b and 6201a are directly synthesized by assigning a 
predetermined overlapping amount, step S1214, Fig 39), (ii) determining the 
relation between the respective coordinates for the pixels in the individual 
cameras and in the synthetic image (the image correction/overlapping amount 
calculator 6016 detects the overlapping regions of the images 6201b' and 6201a' 
by finding correspondences among their pixel values using an algorithm such as 
template matching, step S1214, Fig 39), and (iii) calculating calibration 
parameters from said relation, said calculated calibration parameters being unique 
for the at least two cameras and their current location as related to the object 
being recorded (in step S1215, the image correction/overlapping amount 
calculator 6016 corrects the sensed images, i.e., compensates for the luminance 
and color differences between the two, right and left images that may be produced 
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by the image sensing optical systems 6004b and 6004a, and corrects trapezoidal 
distortion (projection transform), as Fig 41 shows the correction state of 
luminance values that may become discontinuous at the joint portion between the 
right and left parallax images 6201b' and 6201a') (col. 36, line 32 to col. 37, line 

22); 

b. synchronously recording video sequences using each of said at least two video 
cameras (the right and left images are sensed in step S1208 as the right and left 
images that may be produced by the image sensing optical systems 6004b and 
6004a) (col. 37, lines 6-22); and 

c. generating a wide image video sequence by combining said synchronously 
recorded video sequences using said calculated calibration parameters (the 
convergence angle 0lm is calculated based on the depth z detected in step SI 801), 
wherein the step of calculating calibration parameters comprises the step of 
calculating a projective transform by comparing spatial characteristics of said 
corresponding parts with each other, and wherein the step of generating the wide 
image video sequence is performed using said projective transform (in step 

SI 802, the calculated convergence angle 0 is compared with the limit value 0lm 
of the convergence angle held in a memory of each of the image sensing optical 
system drivers, as the method of controlling the image sensing optical systems 
6701b and 6701a to reduce to zero, i.e. performing projective transform, the 
parallax of the principal object 6904, which is located 1 m in front of the image 
sensing optical systems 6701b and 6701a) (col. 43, lines 4-22). 
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Regarding claim 2, the limitations of claim 1 are taught above, Ishikawa discloses the 
synchronously recorded video sequences are stored in a memory means (a memory 6007, Fig 
38) (col. 34, line 55 - col. 35, line 2). 

Regarding claim 3, the limitations of claim 1 are taught above, Ishikawa discloses the 
synchronously recorded video sequences are used concurrently for generating the wide image 
video sequence (moving panoramic image) (col. 35, lines 14-28). 

Regarding claim 5, the limitations of claim 1 are taught above, Ishikawa discloses the 
wide image video sequence (moving panoramic image) is stored on a memory means (a 
memory 6007, Fig 38) (col. 34, line 55 - col. 35, line 2). 

Regarding claim 14, this claim differs from claim 1 only in that the claim 1 is a method 
claim whereas claim 14 recites similar features in an apparatus format. Thus the apparatus 
claim 9 is analyzed and rejected as previously discussed with respected to claim 1 above. 

Regarding claims 15 and 16, these claims recite same limitations as claims 2 and 3, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 2 and 3 above. 

Regarding claim 25, this claim recites same limitations as claim 1. Thus it is analyzed 
and rejected as previously discussed with respect to claim 1 above. 

Regarding claims 26 and 27, these claims recite same limitations as claims 2 and 3, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 2 and 3 above. 

Regarding claim 36, this claim recites substantially the same limitations as claim 1. Thus 
it is analyzed and rejected as previously discussed with respect to claim 1 above. 
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Claim Rejections - 35 USC § 103 

6. The text of those sections of Title 35, U.S. Code not included in this action can be found in a 
prior Office action. 

7. Claims 1-3, 5-8, 10, 14-19, 21, 25-30, 32, and 36 are rejected under 35 U.S.C. § 103(a) as 
being unpatentable over Ishikawa (US 6,549,650) in view of Chang (US 7,307,654). 

Regarding claim 6, the limitations of claim 1 are taught above, Ishikawa does not disclose 
a detailed calibration process according to this claim. However, Chang discloses calculating 
the calibration (calibration process 102, Fig 3); comprises the following steps: 

a. starting of calibration process (step 110, Fig 3) (at 110, a predesigned calibration 
pattern 50 is displayed in front of planar background 18, i.e. on front planar 
surface 44) (4:56-5:11, Chang); 

b. synchronizing the sequences from each of the at least two cameras (blocks 112 
and 116 occur simultaneously, and blocks 1 14 and 118 occur simultaneously or 
near simultaneously) (4:56-5:11, Chang); 

c. computing inter-image projective fransformations (step 122, Fig 3) (the 
correspondence mapping and geometric parameters of planar background 18 
determined at 120 are utilized to compute both internal and external calibration 
parameters of first camera 14 and second camera 16) (5:32-53, Chang); 

d. using the transformations to refer each image to a common reference frame (step 
120, Fig 3) (at 120, CPU 30 compares the robust features from the first and 
second images to the known characteristics of calibration pattern 50 and performs 
a correspondence mapping and the correspondence mapping entails locating each 
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captured robust characteristic of calibration pattern 50 in first image 14 and noting 
the spatial relationship of the captured robust characteristics) (5:12-31, Chang); 

e. selecting a real or virtual reference camera such that certain lines on the pitch or 
stadium arc substantially horizontal and substantially parallel in the wide image 
(step 122, Fig 3) (the overall spatial relationship of first camera 14, second 
camera 16, and planar background 18 is used to determine a first coordinate 
system with respect to first camera 14 and a second coordinate system with 
respect to second camera 16) (5:32-53, Chang); 

f selecting a rectangular region of interest within the wide image (the 

correspondence mapping and geometric parameters of planar background 18, Fig 
1, as the planar background 18 is determined at 124 based upon the calibration 
parameters of each camera) (5:32-53, Chang); and 

g. storing the computed values resulting from the calibration process to be used as 
the calibration parameters (once the two coordinate systems are derived, the 
calibration process is complete and the first subject image or video is recorded by 
first camera 14 and transferred to CPU 30 via first video capture device 32) (5:32- 
6:7, Chang). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Ishikawa's 
device, as the suggestion/motivation would have been to enable the controller which is 
capable of performing the calibration, analysis, and interpolation computations to derive a 
synthesized image, to create synthesized images allows the user to select the most 



Application/Control Number: 10/564,286 Page 8 

Art Unit: 2622 

satisfactory angle to view the scene in order to better ensure that the specific region of 
interest at a particular time instance is accessible to the user (3:29-46 and 10:50-64, Chang). 

Regarding claim 7, the limitations of claims 1 and 6 are taught above, Ishikawa does not 
disclose the step of determining lens distortion parameters for each camera and correcting 
radial distortion in each image. However, Chang discloses the step of determining lens 
distortion parameters for each camera, and correcting radial distortion in each image (adapted 
to stretch the synthesized video image on the video display 20 to remove any parallax 
distortions) (3:47-67, Chang). 

Regarding claim 8, the limitations of claims 1 and 6 are taught above, Chang discloses 
the step of selecting non-linear distortion parameters to reduce perspective distortion of the 
image (the internal calibration parameters include but are not limited to the focal length and 
lens distortion of each camera) (5:32-53, Chang). 

Regarding claim 10, the limitations of claim 1 are taught above, Chang discloses the step 
b (step 114 and 118, Fig 3) is performed automatically by an algorithm for identification of 
corresponding features in concurrent video images and the coordinates for the corresponding 
features are input via a computer means (the second image is transferred from second camera 
16 to CPU 30 via second video capture device 34. CPU analyzes the second image for the 
robust features of calibrated pattern 50) (4:56-5:1 1 and Fig 3, Chang). 

Regarding claims 17, 18, 19, and 21, these claims recite same limitations as claims 6, 7, 
8, and 10, respectively. Thus they are analyzed and rejected as previously discussed with 
respect to claims 6, 7, 8, and 10 above. 
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Regarding claims 28, 29, 30, and 32, these claims recite same limitations as claims 6, 7, 
8, and 10, respectively. Thus they are analyzed and rejected as previously discussed with 
respect to claims 6, 7, 8, and 10 above. 

8. Claim 4 is rejected under 35 U.S. C. § 103(a) as being unpatentable over Ishikawa (US 
6,549,650) in view of Alonso (US 6,445,293). 

Regarding claim 4, the limitations of claim 1 are taught above, Lee and Chang do not 
disclose the wide image video sequence is transmitted live. However, Alonso discloses the 
wide image video sequence is transmitted live (the camera system sets the front camera as 
output image and transmits live video out from this camera) (3:56-65, Alonso). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the imaging pickup device as taught by Alonso into Lee and 
Chang's device, as the suggestion/motivation would have been to guarantee that the camera 
system will remain in the alarm state in case of an emergency (1 :36-39 and 3:41-45, Alonso). 

9. Claims 11,13, 22, 24, 33, and 35 are rejected under 35 U.S.C. § 103(a) as being unpatentable 
over Ishikawa and Chang, and further in view of Matsumoto (US 2003/0071906). 

Regarding claim 11, the limitations of claim 1 are taught above, Chang discloses the 
method according to claim 1 which comprises the following steps: 

a. apply the calculated calibration parameters (at 1 10, a predesigned calibration 
pattern 50 is displayed in front of planar background 18, i.e. on front planar 
surface 44) (4:56-5:1 1 and step 1 10, Fig 3, Chang); 
c. retrieving one new image from each camera (at 112, first camera 14 captures a 
first image of calibration pattem 50 and at 1 16, second camera 16 captures a 
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second image of calibration pattem 50) (4:56-5:1 1 and steps 1 12 and 1 16, Fig 3, 
Chang); 

d. selectively updating the parameters needed to transform intensities in one or more 
of the cameras to eliminate visible seams (correspondence mapping entails 
locating each captured robust characteristic of calibration pattem) (5:12-31 and 
step 120, Fig 3, Chang); 

e. selectively adjusting intensities in the images from one or more of the cameras (a 
similar correspondence mapping procedure is completed using the second image 
to determine the geometric parameters of planar background 18 with respect to 
second camera 16) (5:12-31 and step 120, Fig 3, Chang); 

f. creating the current seamless, wide image from the current images from each 
camera (map corresponding robust features to determine geometric parameters of 
planar background, since the calibration pattem 50 is a planar surface located 
upon planar backgromd 18, the geometric parameters of planar background with 
respect to first camera 14 are directly computed from the mapped correspondence 
information) (5:12-31 and step 120, Fig 3, Chang); and 

g. outputting the image to a display or to a memory storage area (storage device 37, 
Fig 1) (after all calibration parameters are determined, the overall spatial 
relationship of first camera 14, second camera 16, and planar background 18 is 
determined at 124 based upon the calibration parameters of each camera and the 
image is to a display or to a memory means) (5:32-53, Chang). 



Application/Control Number: 10/564,286 Page 11 

Art Unit: 2622 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Ishikawa's 
device, as the suggestion/motivation would have been to enable the controller which is 
capable of performing the calibration, analysis, and interpolation computations to derive a 
synthesized image, to create synthesized images allows the user to select the most 
satisfactory angle to view the scene in order to better ensure that the specific region of 
interest at a particular time instance is accessible to the user (3:29-46 and 10:50-64, Chang). 

Ishikawa and Chang do not disclose a calibration process is repeated until the end of the 
sequence is reached or return to step b until end of generation of the video sequence. 
However, Matsumoto disclose a calibration process is repeated until the end of the sequence 
is reached (the calibration process is repeated at a predetermined interval until the release 
switch (SWl) 126 or the release switch (SW2) 127 is pressed. Figs 7-9) ([0183], Matsumoto) 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Matsumoto into Ishikawa 
and Chang's device, as the suggestion/motivation would have been to enable that the more 
accurate correction data can be obtained ([0183], Matsumoto). 

Regarding claim 13, the limitations of claims 1 and 11 are taught above, Chang discloses 
the new images from each video camera are read from a memory means (storage device 37, 
Fig 1) (after all calibration parameters are determined, the overall spatial relationship of first 
camera 14, second camera 16, and planar background 18 is determined at 124 based upon the 
calibration parameters of each camera and the image is to a display or to a memory means) 
(5:32-53, Chang). 



Application/Control Number: 10/564,286 Page 12 

Art Unit: 2622 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the calibration process as taught by Chang into Lee's device, 
so as the image data (e.g., video images, image shape and color information) stored in 
storage device 37 will be available for use at a later time (6:63-7:20 Chang). 

Regarding claims 22 and 24, these claims recite same limitations as claims 1 1 and 13, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 11 and 13 above. 

Regarding claims 33 and 35, these claims recite same limitations as claims 1 1 and 13, 
respectively. Thus they are analyzed and rejected as previously discussed with respect to 
claims 11 and 13 above. 
10. Claims 12, 23 and 34 are rejected under 35 U.S.C. § 103(a) as being unpatentable over 
Ishikawa, Chang and Matsumoto, and further in view of Alonso (US 6,445,293). 

Regarding claim 12, the limitations of claims 1 and 1 1 are taught above, Ishikawa, Chang 
and Matsumoto do not disclose the new images from each camera are read from live sources, 
each such source comprising a video camera. However, Alonso discloses the new images 
from each camera are read from live sources, each such source comprising a video camera 
(the camera system sets the front camera as ou^ut image and fransmits live video out from 
this camera) (3:56-65, Alonso). 

Thus it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to include the imaging pickup device as taught by Alonso into Ishikawa, 
Chang and Matsumoto 's device, so as the camera system will be capable to used by the main 
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system to increase system versatility and can get any advanced feature shown in the main 
video system (3:41-45, Alonso). 

Regarding claims 23 and 34, these claims recite same limitations as claim 12. Thus they 
are analyzed and rejected as previously discussed with respect to claim 12 above. 



Conclusion 

1 1 . The prior art made of record and not rehed upon is considered pertinent to applicant's 
disclosure: 

- Katayama et al. (US 7,098,914) provide an image synthesis method, an image 
synthesis apparatus, and a storage medium in which a mapping mode is easily set 
when synthesizing an image; 

- Hubel et al. (US 6,930,703) provide a method and apparatus for automatically 
capturing a plurality of images during a pan; 

- Murata et al. (US 7,3 13,289) disclose an image processing method that effectively 
corrects the image distortions caused by oblique imaging in which an original image 
on the object plane is taken from an oblique direction, and provides an improved 
operability for the user; and 

- Colavin et al. (US 7,064,783) disclose a still picture format for subsequent picture 
stitching for forming a panoramic image. 
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Inquiries 

12. Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Kent Wang whose telephone number is 571-270-1703. The examiner 
can normally be reached on 8:00 A.M. - 5:30 PM (every other Friday off). 
Sinh Tran can be reached on 571-272-7564. The fax phone number for the organization 
where this application or proceeding is assigned is 571-270-8300. 

Information regarding the status of an apphcation may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status information 
for unpublished applications is available through Private PAIR only. For more information 
about the PAIR system, see http://portal.uspto.gov/extemal/portal/pair. Should you have 
questions on access to the Private PAIR system, contact the Electronic Business Center 
(EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer 
Service Representative or access to the automated information system, call 800-786-9199 (IN 
USA OR CANADA) or 571-272-1000. 
/TUAN HO/ 

Primary Examiner, Art Unit 2622 
KW 
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